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ABSTRACT 

A computer-assisted r diagnostic assessment system 
(Level III) is described that has been implemented on an IBM 370/155 
and is written in Coursewriter III, Version 3. The function of this 
system is to administer a series of 10 examinations to 64 first year 
medical students. The system fulfills the University of Illinpis 
School of Medicine »s assessment goals by providing a diagnostic, 
nongraded system which furnishes students with continuous feedback on 
thexr progress. Students have responded favorably to tjieir Level III 
experience. They especially appreciate the immediacy of the feedback 
after each exa^d (printout) and the complete listing and reference 
information given for each uncorrected question. In addition tq 
availability through Coursewriter systems, a version of Level III is 
being prepared for use on the PLATO IV computer system. (WCM) 
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Introduction 



This paper dcscribas a coniputer~asqi«?f ^>r? A-t.r, 
(Level III) that has been l.pleLnted on anleM 37^^^^^^ ^^^^^^"^ 
Coursewritor III, Version 3. The fm^ctLn Ik written in 

series of 10 eKa.inatioas to eA^fi^sry^:" idL'^l llTn.i: " '^'^'"'^^^^ ^ 
Backgr ouod 

^r,.^^^'^^^: l^l- sconces, ' 

physicians had revised the conventional h.!?. J ' ^''^'^^^ty and local 
beslnn.og ^.dicnl .tudont. ^^r^^^ .'^ nf 'rf'""!""^ to provide 
would allow a student to entor n ^ ^ ^kill«; .-^nd knov1-r»-« t-.h^t 

poratc. ovar 2,000 specific Larning oM^otlves 'n lfs,= J'""' 

u^JJuf: If :ajj^i J^co™:- L\^=:ir! °' - -rj:r?!3^i.-'- 

.vaX„a.o., .°r?,--L-i'^^:.L1„7:~ a^^a^-:LV.„e3. 

in tho%'L\\.Jl;\,lJr98^'':ru'-r:;ol"f'"' yoar st„de„ts have participated 
IlUnola first year colrehenslvf ^v'^ successfully passing the University oC 
Board Part I ex'aination <=-™iuatIo„, and 8« passing the National 

ITie Role df Studon^ Assosstien t 
based\VL'aJnJnrote"ti:^^^^ 

immediate feedback on the ir^e^'oma, le^ r ^^"^^^'^t and 

is built into the SBMS-UC cu!:ri;.^irirthr o a.l^s^' n^-^J-J^- -f P^^^-ss 
student via the pre- and post-test om4ionr^f I, of " r ^ s^lf testing by the 
the 360 learning units- 2) ohl^r^ J • ^ '"^'^ each of 

^ henslve examination UvoJ m) , i' ' oTr''"'^ -^"-oss^eat ulth a co.pre- 

0 curriculum divisions (cHnio-.JnriM \ ^'^P^'^'^'^ after each of the 10 

X ticing physlclL aeve J^oJ the ul^t^ 2ulf ^valuation by a prac- 

\J sciences to the clinical ptoblem. ^^"^ 



levol III Phtloso p}\y and Objecttvns 



The I.cvfil III examination is a cumulative examination covering all the 
basic science units included in n clinical problem. Currently, 12 basic 
science disciplines are included in the examinations; anatomy, behavioral 
science, biochemistry, embryology* genetics, histology, irmnunology, micro- _ 
biology, neuroscicnce, pathology, pharmacology, and physiology. Kach Level 
III examination includes approximately 180 questions and takes hours 
for a student to complete. Kach Level III question is based on instructional 
objectives which are included within each curriculum unit. The primary goal 
of the Level III student assessment procedure is to provide a meaningful 
learning experience for each student. 

Tlie objectives of> the Level III assessment system are as follows: 

A. The Level III is an instrument to improve learning, not to assign 
grades. During the administration of each exam, the student is 
recycled back to each incorrect response and is given additional 
opportunities to achieve the correct answer, The student should 
leave each questioa with a positive experience, i.e. the correct 
answer. In a few words, the exam should be informative and not 
punitive in nature, 

B. Each Level III is based on instructional objectives which are 
included v?ithin each learning unit. * 

C. Each Level III provides periodic testing which motivates students 
by providing them with short-term goals toward which to work, by 
clarifying for students what learning outcomes' are expected^ and 
by providing them with feedback concerning their progress. 

D. Each Level III directs student learning efforts toward the objec- 
tives being measured. For this reason, each Level III is devised 
to test achievements for a representative sample of objectives of 
each learning unit. 

E. . Each Level III is diagnostic because it reveals the learning 

weaknesses of individual students in each curriculum unit. 
This is provided by listing for the student on a printout what 
questions he has missed. Also, through a detailed item analysis 
the com^^osite results will provide the School with precise infor- 
mation on curriculum deficiencies. 

F. Each Level III provides immediate feedback, thus allowing students 
to assess their progress, and to take corrective action inimcdiately, 

* This feedback also provides their advisors a constant and up--to- 
date picture of each student's progress and deficiency. 

' Rationale for Computo r-Assi stance 

Computers-Assisted Instruction (CAl) was turned to as a medium which 
could facilitate tlie desired flcKibility and achieve the above-mentioned 
objectives. First of all, the interactive nature of CAT would facilitate 



the attempt to provide a ''toachlng cxnm". It would also allow some form of 
tutoring fcodbnck (doaling with specific questions during the examination) 
tailored to the responses made by the student, Examinations could be pro- 
cessed Individually and scared automatically with summarized information 
printed out in a matter of minutes. The nature of this summary information 
could be designed at will to meet tie diagnostic needs of both the student 
and his advisors^ 

Systems Hardware and Software , . 

The computer program (named LEVELS) which handles this process Is 
written in Courscwriter III, Version 3, as implemented on an IBM 370/155 
operated by the University of Illinois Medical Center In Chicago. Terminals 
are connected by means of acoustic couplers and a telephone tie line between 
the Urbana and Chicago campuses. For student Interaction, three Hazeltlne 
2000 video-display terminals are used. For hard copy output, an IBM Com- 
municating Mag Card terminal is used. • 

" > , * ■ • 

Stud ent Use 

3 

Figure 1 summarizes the flow in LEVEL3 as seen by the student . When 
the student first signs on to LEVELS, he is presented with an explanation of 
the use of the program (scoring, skipping questions, erasing, etc.) and a 
list of clinical problem examinations from which to choose (Fig. 1|A). 
Having chosen an exam> he is presented with a choice of disciplines tested 
in that exam, along with the number of questions for each discipline (Fig. 
1, B). The choice o£ a discipline branches the student to a linear block of 
questions. As the student moves through this block, the computer accepts 
and scores his answers without feedback. In addition to giving an ansv/er 
which may be scored, the student may enter other responses which give hlra 
some control of the program (Fig, 1,C). Entering a blank answer allows the 
student to choose not to answer and receive a zero score on the question. 
Responding ^'skip" (Fig, 1,D) allows the student to skip over a question with 
out It being scored and be returned to it after he has seen the remaining 
questions in the section. At the end of a discipline, the program gives the 
student h'is score for the discipline and allows him to review the questions 
that the student missed. . 

During this review, the student is given the opportunity to correct his 
errors. If he is successful, bonus points are added to his score (an incen- 
tive to correct errors) ; if not , the program holds him on the question 
coaching him toward the correct answer. He may also request the correct 
answer from the computer (Fig. 1,E). When the student finishes correcting a 
section, he is given his '^raised score" and is allowed to choose another 
discipline, l^lien the student finishes all of the disciplines, he sign$ off 
and receives a hard copy printout of his results within 30 minutes (Figure 
2), This printout includes the student *s scores and a listing (with text- 
book references) of all questions which he was unable to. correct. 

The student, upon receiving his exam printout^ may check out a paper 
copy of the exam on a "secure basis" and use these as learning tools. Armed 
with the bottom half of the printout, the paper copy examination, and his 
textbooks the student should proceed to understand all the correct answers. 



Figure 3 Flow Chart of LEVEL3 Interaction 
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Figure 2 - Computer Printout 



examination: coronary heart disease 



running time of exam: 4 hr. 7 mln. 
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Grant, p. 478 

Woodburne, p. 323 

P&H p. 274; Netter V, p. 253 

Yalom 3-14 
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Lehnlnger, pp. 348-349 

Lehnlnger, p. 182 
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R & A, pp. 182-184 
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Scor ing of Quos tlons 



LEVEL3 Is deslp,nGcl to allow a wide variety of multiple-choice typo 
questions and to niakc the Input of new questions and revision of old ones as 
simple as possible. 

Each question may have any number of. choices correct rather than the 
one choice correct per question format used on most standarized exams. The 
students are informed that one or more choices for each question may be 
correct, thus leaving it to the student to identify the proper combination 
of choices. The scoring system also allows for partial credit wiien the 
completely correct answer is a combination of the choices offered and the 
student identifies some but not all oi: them. Thus, each question is scored 
on a seml-^continuous scale from --1 to +1 according to the following formula . 

If of stude nt' s choic es co rrect _ // of student ^s choices wron g 
p of possible correct choices // of possible wrong choices 

In a situation where all choices are correct, the second term of the 
formula is dropped and the scale of scoring is from 0 to HI* 

Item Analysis . ^ 

By allowing the student to enter all answers which he thinks are correct 
to an item, detailed performance data is provided on each specific response 
within each question. This question- performance data is then recycled back 
to the facul{:y and into the curriculum for modification of objectives or 
learning experiences. 

Co nclusion 

The LEVEL3 computer-assisted program has fulfilled the School's assess- 
ment goals by providing a diagnostic, nongraded system which furnishes 
students continuous feedback on their progress. Students have responded 
favorably to their Level III experience throughout the year. They espec- 
ially appreciate the immediacy of the feedback after each exam (printout) 
and the complete listing and reference information given for each uncor- 
rected question. In addition to availability through Coursewr iter systems, 
a version of LEVEL3 is being prepared for use on the PLATO IV computer 
system. 
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